Apparatus and method for converting signals

ABSTRACT

An image-coding format converting apparatus comprising an EMPEG2 image decoder  30 , a resolution/frame rate converter  31 , a motion vector converter 32, and an MPEG4 encoder  33 . The EMPEG2 decoder  30  decodes a bit stream of EMPEG2 image codes, generating an image signal. The resolution/frame rate converter  31  converts the image signal. The MPEG4 encoder  33  encodes the output of the resolution/frame rate converter  31 , generating a bit stream of MPEG4 image codes. In the resolution/frame rate converter  31 , pixels are added or extracted in accordance with the start position of a macro block, thus adjusting the input image signal to one than can easily be encoded to MPEG4 image codes. The motion vector converter  32  generates an MPEG4 motion vector from parameters such as an MPEG2 motion vector. The MPEG4 encoder  33  uses the MPEG4 motion vector to encode the output of the resolution/frame rate converter  31.

BACKGROUND OF THE INVENTION

The present invention relates to an apparatus and method for convertingsignals. More particularly, the invention relates to the technique ofconverting a signal encoded in the first format, into a signal encodedin the second format.

FIG. 1 shows a conventional apparatus for converting an image-codingformat. The apparatus is designed to convert an input bit stream ofimage codes into a bit stream of a different image-coding format. Morespecifically, the image-coding format converting apparatus of in FIG. 1converts an input bit stream of MPEG2 (Moving Picture image codingExperts Group: ISO/IEC13818-2) image codes, into a bit stream of MPEG4(ISO/IEC14496-2) image codes.

In the image-coding format converting apparatus shown in FIG. 1, aninput bit stream of MPEG2 image codes is supplied to the MPEG2 imagedecoder 210.

The MPEG2 image decoder 210 decodes the input bit stream of MPEG2 imagecodes, in accordance with the MPEG2 image-decoding scheme, thusgenerating an image signal. The image signal is input to theresolution/frame rate converter 211.

The resolution/frame rate converter 211 converts the image signal to animage signal that has different resolution and frame rate. The signalthe converter 211 has generated is supplied to an MPEG4 image encoder212. It should be noted that the converter 211 reduces the resolution ofthe decoded image signal to half the original value in both the verticaldirection and the horizontal direction.

The MPEG4 image encoder 212 encodes the image signal supplied from theresolution/frame rate converter 211, in accordance with the MPEG4image-encoding scheme, thereby generating an MPEG4-encoded bit stream.The MPEG4-encoded bit stream is output from the image-coding formatconverting apparatus of FIG. 1.

The MPEG4 image encoder 212 detects a motion vector from the imagesignal generated by the MPEG2 image decoder 210 and predict a motionfrom the motion vector, in the same way as in the ordinary encodingprocess.

The amount of data processed to detect the motion vector in the courseof encoding the input image signal occupies about 60 to 70% of all dataprocessed in the conventional method of converting an image-codingformat. Consequently, it is difficult to process the image signal inreal time. This will inevitably result in a time delay. Further, theimage-coding format converting apparatus needs to be large and complex.

In the image-coding format converting apparatus described above, theresolution/frame rate converter 211 changes the resolution and framerate of the image signal that the MPEG2 image decoder 210 has generatedby decoding the bit stream of MPEG2 image codes in accordance with theMPEG2 image-decoding scheme. Then, the MPEG4 image encoder 212 encodesthe image signal thus processed by the converter 211, in accordance withthe MPEG4 image-encoding scheme, and generates an MPEG4-encoded bitstream.

Therefore, the resolution and frame rate of the image signal output fromthe resolution/frame rate converter 211 may not be appropriate, makingit difficult for the MPEG4 image encoder 212 to perform MPEG4 imageencoding correctly.

BRIEF SUMMARY OF THE INVENTION

The present invention has been made in view of the above. An object ofthe invention is to provide an apparatus and method for convertingsignals, in which at least a resolution-converting process is performedon a signal encoded in the first format, such as a bit stream of MPEG2image codes, thereby generating a second signal, and a third signal,such as a bit stream of MPEG2 image codes, is generated from the secondsignal. The third signal thus generated preserves the quality of thefirst signal. The third signal can be generated by processing a smallamount of data. The apparatus can therefore be relatively small andsimple in structure.

An apparatus according to the invention converts a first encoded imagesignal to a second encoded image signal. The apparatus comprises:signal-decoding means for decoding the first encoded image signal,thereby outputting an image signal and a first motion vector used indecoding the first encoded image signal; signal-converting means forconverting the image signal to a second image signal; motion-vectorconverting means for converting the first motion vector for the secondimage signal, thereby outputting a second motion vector; andsignal-encoding means for encoding the second image signal into thesecond encoded image signal, by using the second motion vector.

An apparatus and a method according to the invention performs at leastresolution conversion on a first signal, thereby generating a secondsignal, and for generating a third signal from the second signal. Thisapparatus comprises: parameter-converting means for converting processparameters contained in the first signal to all or some of parametersthat are required to generate the third signal from the second signal;and signal-generating means for generating the third signal from thesecond signal by using the parameters generated by theparameter-converting means. The apparatus can therefore achieve theobject of the invention.

A method according to this invention converts a first encoded imagesignal to a second encoded image signal. The method comprises the stepsof: decoding the first encoded image signal, thereby outputting an imagesignal and a first motion vector used in decoding the first encodedimage signal; converting the image signal to a second image signal;converting the first motion vector for the second image signal, therebyoutputting a second motion vector; and encoding the second image signalinto the second encoded image signal, by using the second motion vector.

A method according to the present invention performs at least resolutionconversion on a first signal, thereby generating a second signal, andfor generating a third signal from the second signal. The methodcomprises the steps of: converting process parameters contained in thefirst signal to all or some of parameters that are required to generatethe third signal from the second signal; and generating the third signalfrom the second signal by using the parameters generated in the step ofconverting the process parameters.

In the apparatus according to the present invention, at least resolutionconversion is performed on a first signal, thereby generating a secondsignal, and a third signal is generated from the second signal. Theprocess parameters contained in the first signal are converted to all orsome of the parameters that are required to generate the third signal.Hence, the amount of data that should be processed to generate the thirdsignal from the second signal can be reduced, without deteriorating thequality of the third signal. This prevents the apparatus from becominglarge and complex.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1 is a block diagram showing a conventional apparatus forconverting an image-coding format;

FIG. 2 is a block diagram illustrating an apparatus for converting animage-coding format, which is the first embodiment of the presentinvention;

FIG. 3 is a diagram explaining the correlation between the motionvectors an image has before and after its resolution is changed;

FIG. 4 is a diagram explaining how the resolution and frame rate of animage signal are converted;

FIG. 5 is a diagram explaining how pixels are added or removed from animage in accordance with an image-size adjusting flag;

FIG. 6 is a diagram illustrating the principle of converting a motionvector;

FIG. 7 is a block diagram showing a motion vector converter andexplaining the operation of the motor vector converter;

FIG. 8 is a diagram explaining in detail how the vector converter;

FIG. 9 is a diagram illustrating the concept of the motion vectorconversion that the vector converter performs when the inter-macro blockhas a frame structure and should be subjected to frame prediction;

FIG. 10 is a diagram depicting the concept of the motion vectorconversion that the vector converter effects to predict a top field;

FIG. 11 is a diagram depicting the concept of the motion vectorconversion that the vector converter effects to predict a bottom field;

FIG. 12 is a diagram showing how the positional relation between fieldschanges with time;

FIG. 13 is a diagram explaining in detail how a motion-vector adjustingsection operates;

FIG. 14 is a diagram explaining in detail how a motion vector correctoroperates;

FIG. 15 is a block diagram showing the second embodiment of thisinvention; and

FIG. 16 is a block diagram illustrating the third embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

Embodiments of the present invention will be described, with referenceto the accompanying drawings.

FIG. 2 shows an apparatus for converting an image-coding format, whichis the first embodiment of the invention.

In the apparatus shown in FIG. 2, an input bit stream of MPEG2 imagecodes is supplied to the MPEG2 image decoder 30.

The MPEG2 image decoder 30 decodes the bit stream in accordance with theMPEG2 image-decoding scheme, thus generating an image signal. The imagesignal (an interlace image signal) is input to the resolution/frame rateconverter 31.

The resolution/frame rate converter 31 converts the image signal to animage signal that has a given resolution and a given frame rate. Theconverter 31 adjusts the resolution of the image signal in accordancewith an image-size adjusting flag supplied from an external device.Thus, the section 31 outputs an image signal having such a resolutionthat the image signal may be converted into an MPEG4 image signal. Theimage signal the resolution/frame rate converter 211 has generated issupplied to the MPEG4 image encoder 33.

The MPEG4 image encoder 33 encodes the image signal supplied from theresolution/frame rate converter 31, thereby generating an MPEG4-encodedbit stream. The MPEG4-encoded bit stream is output from the image-codingformat converting apparatus (FIG. 2).

This is how the apparatus shown in FIG. 2 operates. The image-codingformat converting apparatus operates, utilizing the fact that the motionvector of an image greatly changes in terms magnitude and directionafter the image is subjected to resolution/frame rate conversion. Thatis, the motion vector of MPEG2 system (hereinafter referred to as “MPEG2motion vector”), which the MPEG2 image decoder 30 uses to decode theinput bit stream, is converted to a motion vector of MPEG4 system(hereinafter referred to as “MPEG4 motion vector”). The MPEG4 motionvector, thus acquired, is supplied to the MPEG4 image encoder 33. Usingthe MPEG4 motion vector, the MPEG4 image encoder 33 encodes the imagesignal supplied from the resolution/frame rate converter 31. It istherefore unnecessary for the encoder 33 to detect motion vectors.

More specifically, the image-coding format converting apparatuscomprises a motion vector converter 32. The motion vector converter 32receives the MPEG2 motion vector the MPEG2 image decoder 30 has used todecode the input bit stream. The converter 32 converts the MPEG2 motionvector to an MPEG4 motion vector, which is supplied to the MPEG4 imageencoder 33. The MPEG4 image encoder 33 uses the MPEG4 motion vector,thereby encoding the image signal generated by the resolution/frame rateconverter 31.

As described above, the MPEG4 image encoder 33 need not detect motionvectors in the image-coding format converting apparatus according to thepresent invention. This reduces the amount of data the apparatus mustprocess and prevents the apparatus from becoming large or complex. Sincethe apparatus needs only to process a small amount of data, the timedelay is small enough to achieve real-time processing of image data.

In the present embodiment, or image-coding format converting apparatus,not only the motion vector, but also parameters such as the image sizeand macro-block type, both the MPEG2 decoder 30 uses to decode the inputbit stream, or the parameters obtained after the resolution and framerate are converted, are applied in the MPEG4 encoder 33 to generateMPEG4 image codes. The amount of data that the MPEG4 image encoder 33must process is thereby decreased. Hence, the encoding efficiencyfurther can increase and the process time can reduce.

The correlation between the motion vectors an image has before and afterits resolution is changed will be explained, with reference to FIG. 3.Shown in FIG. 3A is a frame image before the resolution conversion.Shown in FIG. 3B is the frame image after the resolution conversion. Theregion dm in each frame image indicates the position that the image ofan object ob assumes in the frame image before the resolutionconversion. The image of the object ob before the resolution conversionhas a motion vector MV shown in FIG. 3A. The image of the object obafter the resolution conversion has a motion vector MV shown in FIG. 3B.

The horizontal component of the motion vector MV the image has after theresolution conversion can be obtained from two items of data, i.e., thehorizontal component of the motion vector MV the image has before theresolution conversion, and the rate at which the resolution of the imagehas been converted in the horizontal direction. The vertical componentof the motion vector MV the image has after the resolution conversioncan be calculated from two items of data, too, i.e., the verticalcomponent of the motion vector MV the image has before the resolutionconversion, and the rate at which the resolution of the image has beenconverted in the vertical direction. Namely, the motion vector MV afterthe resolution conversion is correlated with the motion vector MV beforethe resolution conversion are related. Only if this correlation isutilized, the vector MV after the resolution conversion can bedetermined from the vector MV before the resolution conversion.

Thus, in the image-coding format converting apparatus of this invention,the motion vector of a macro-block of MPEG2 image codes contained in theinput bit stream or the parameters such as the macro-block type, aresupplied to the motion vector converter 32 in order to convert the inputbit stream of MPEG2 image codes to a bit stream of MPEG4 image codes.The motion vector converter 32 converts the motion vector to an MPEG4motion vector, or the parameters to parameters such as a macro-blocktype. The term “macro-block type” means a code that designates theencoding method for each macro-block.

Referring back to FIG. 2, the image-coding format converting apparatusof this invention will be described in detail.

The MPEG2 image decoder 30 decodes the variable-length MPEG2 image codesof the input bit stream, thereby generating an interlace image signal.The interlace image signal is supplied to the resolution/frame rateconverter 31.

The resolution/frame rate converter 31 comprises a resolution/frameconverter 34 and a pixel adder/extractor 35. The resolution/frameconverter 34 performs resolution conversion and frame-rate conversion.The pixel adder/extractor 35 adds pixels to the image signal or extractpixels therefrom.

The resolution/frame converter 34 converts the resolution and frame rateof the interlace image signal supplied from the MPEG2 image decoder 30.In the present embodiment, the converter 34 reduces the resolution ofthe input signal to half, in both the vertical direction and thehorizontal direction.

More precisely, the resolution/frame converter 34 extracts either topfields or bottom fields from the input interlace image signal as isillustrated in FIG. 4, converting the image signal to a signalrepresenting a progressive image. The resolution of the input imagesignal is thereby decreased in the horizontal direction, to half theoriginal value. Further, the resolution/frame converter 34 converts theresolution of the input signal in the vertical direction, to half theoriginal value, by using a down-sampling filter. To impart a low bitrate to the image signal, the resolution/frame converter 34 extracts anI picture (intra-image coded picture) and a P picture (forwardprediction coded picture) from the image signal, leaving the B pictures(bi-direction prediction coded pictures). That is, the converter 34compresses the image signal in the direction of time (thus, lowering theframe rate). As shown in FIG. 4, the input interlace image is composedof top fields and bottom fields. Of these fields, only the top fieldsare extracted, forming a progressive image. The resolution of the imagetherefore decreases to half in the vertical direction. The progressiveimage constituted by top fields only is subjected to down sampling,thereby extracting only the I picture and the P picture. The frame rateis thereby reduced. The down sampling may be carried out after thereduction of the frame rate. That is, the resolution/frame converter 34changes the resolution and frame rate of the input MPEG2 image signalcomposed of I, B, B, P, B, B, P, . . . pictures, thereby converting theinput image signal to an image signal composed of I, P, P, P, P, . . .top fields only. The output of the resolution/frame converter 34 issupplied to the pixel adder/extractor 35.

The pixel adder/extractor 35 processes the image signal subjected to theresolution/frame rate conversion, so that the MPEG4 image encoder 33 mayencode the image signal to a bit stream of MPEG4 image codes. To be morespecific, the pixel adder/extractor 35 changes the input image data toone consisting of pixels that are arranged in rows and columns, eitherbeing a multiple of 16. To generate such image data, the pixeladder/extractor 35 adds pixels to, or extract pixels from, the inputimage signal, in accordance with an image-size adjusting flag suppliedfrom the external device. The image-size adjusting flag is supplied fromthe external device to the resolution/frame rate converter 31. In theconverter 31, the flag is input to the pixel adder/extractor 35. Theflag causes the pixel adder/extractor 35 to add pixels to, or extractpixels from, the input image signal, if the pixels constituting theimage signal are arranged in rows and columns, either not being amultiple of 16. Hence, if the image-size adjusting flag designatesaddition of pixels or extraction of pixels, the pixel adder/extractor 35adds pixels to, or extracts pixels from, the image signal, changing theimage signal to one that consists of pixels are arranged in rows andcolumns, either being a multiple of 16.

It will be described how the pixel adder/extractor 35 adds or extractspixels in accordance with the image-size adjusting flag, with referenceto FIG. 5.

Assume the image signal G1 output from the MPEG2 image decoder 30consists of m×n pixels, where m and n are multiples of 16. Theresolution/frame converter 34 reduces the resolution of the image signalto half, in both the vertical direction and the horizontal direction,thus generating an image signal G2. The image signal G2 thereforeconsists of m/2×n/2 pixels. When m/2 and n/2 are divided by 16, theremay remain eight pixels.

If m/2 and n/2 are both multiples of 16, the image signal G2 representsan image that can be processed into a bit stream of MPEG4 image codes.In this case, the pixel adder/extractor 35 need not add pixels to orextract pixels from the image signal G2. If m/2 or n/2 is not a multipleof 16 (that is, if eight pixels remain when m/2 or n/2 is divided by16), the image signal G2 represents an image that cannot be processedinto a bit stream of MPEG4 image codes. If so, the pixel adder/extractor35 needs to add pixels to or extract pixels from the image signal G2.

Therefore, the external device supplies the image-size adjusting flag tothe pixel adder/extractor 35, selecting either addition of pixels orextraction of pixels (SW1). In the pixel adder/extractor 35 it isdetermined whether both m/2 and n/2 are multiples of 16 or not (StepJ1). In accordance with the decision made, it is then determined whetherthe image-size adjusting flag should be used or not.

If NO in Step SJ1, that is, if it is determined that both m/2 and n/2are not multiples of 16 and there remain eight pixels, after theresolution/frame converter 34 converted the image signal, and if theimage-size adjusting flag input designates extraction of pixels, thepixel adder/extractor 35 extracts the remaining eight pixels that form afirst or the last row or column. Thus, the pixel adder/extractor 35generates an image signal G3 that consists of (m/2−8)×(n/2) pixels or(m/2)×(n/2−8) pixels.

If NO in Step SJ1, that is, if it is determined that both m/2 and n/2are not multiples of 16 and there remain eight pixels, after theresolution/frame converter 34 converted the image signal, and if theimage-size adjusting flag input designates addition of pixels, the pixeladder/extractor 35 copies eight pixels copied from the original imagedata and adds the copied pixels to a first or the last row or column. Inthis case, the pixel adder/extractor 35 generates an image signal 4 thatconsists of (m/2+8)×(n/2) pixels or (m/2)×(n/2+8) pixels.

No matter whether pixels are extracted from or added to the imagesignal, the pixel adder/extractor 35 outputs an image signalrepresenting an image that consists of m×n pixels, where m and n areboth multiples of 16. The image signal G4 represents an image that canbe encoded into a bit stream of MPEG4 image codes.

The image signal, which the resolution/frame rate converter 31 hasgenerated by performing resolution conversion and pixel addition orextraction, is input to the MPEG4 image encoder 33.

In the meantime, the motion vector of a macro-block generated by theMPEG2 decoder 30 and composed of only P pictures or the parameters suchas the macro-block type are supplied to the motion vector converter 32.

The motion vector converter 32 converts the motion vector supplied fromthe MPEG2 decoder 30 to an MPEG4 motion vector, or the parameterssupplied from the MPEG2 decoder 30 to parameters such as an macro-blocktype.

The principle of the motion-vector conversion the motion vectorconverter 32 effects will be explained, with reference to FIG. 6.

The image signal G1 is shown in FIG. 6A. The image signal G2 is shown inFIG. 6B. Each square defined by solid lines indicates a macro block. Theimage signal G1 shown in FIG. 6A has been output from the MPEG2 decoder30. (That is, the signal G1 has yet to be subjected to resolutionconversion.) The image signal G2 shown in FIG. 6B is generated by theresolution/frame rate converter 31, which decreases the resolution ofthe image represented by the signal G1 to half the original value, inboth the vertical direction and the horizontal direction.

Of the image signal G1 shown in FIG. 6A, which will be subjected toresolution conversion, the upper-left 16×16 pixel macro block will beconverted to the upper-left 8×8 macro block of the image signal G2 shownin FIG. 6B. More precisely, the resolution/frame rate converter 31decreases the resolution of the image signal G1, i.e., m×n pixels (m andn are both multiples of 16), to half the original value in both thevertical direction and the horizontal direction. The image signal G1 isthereby converted to the image signal G2 that represents an imagecomposed of m/2×n/2 pixels as illustrated in FIG. 6B. In this case, fourmacro blocks MB1 to MB4, included in the image signal G1 and eachcomposed of 16×16 pixels, are converted to four blocks b1 to b4 includedin the image signal G2 and each composed of 8×8 pixels. The four blocksb1 to b4 included in the image signal G2 constitute one macro blockMB_(T).

The macro blocks MB1 to MB4, included in the image signal G1 and eachcomposed of 16×16 pixels, have motion vectors MV1 to MV4. Similarly, themacro blocks b1 to b4, included in the image signal G2 and each composedof 8×8 pixels, have motion vectors mv1 to mv4. The motion vectors MV1 toMV4 are greatly correlated with the motion vectors mv1 to mv4. Themotion vector converter 32 can therefore effectuate motion-vectorconversion T1, finding the motion vectors mv1 to mv4 of the blocks b1 tob4 of the image signal G2, from the motion vectors MV1 to MV4 of themacro blocks MB1 to MB4 included in the image signal G1. Moreover, theconverter 32 can obtain the motion vector MV_(T) of the macro blockMB_(T) composed of the four blocks b1 to b4, from the motion vectors mv1to mv4 of the blocks b1 to b4.

The motion vector converter 32 that converts motion vectors as indicatedabove will be described in respect of its structure and operation, withreference to FIG. 7.

As FIG. 7 shows, the motion vector converter 32 comprises a vectorconverter 70, a motion vector adjuster 71, an operation section 72, anda motion vector corrector 73. The vector converter 70 receives themotion vector MV of a macro block MB composed of 16×16 pixels, which hasbeen output from the MPEG2 decoder 30 and which can be processed intoMPEG3 image codes. The vector converter 70 receives parameters such asthe image size, macro-block type and the like, too.

The vector converter 70 generates a motion vector from the motion vectorMV of a macro block MB composed of 16×16 pixels and the parameters suchas the image size, macro-block type and the like. The motion vector thusgenerated corresponds to a block composed of 8×8 pixels, the resolutionand frame rate of which have been converted.

How the vector converter 70 operates will be described in detail, withreference to FIG. 8. The vector converter 70 processes the motion vectorMV of a macro block MB composed of 16×16 pixels and the parameters suchas the image size, macro-block type and the like, as will be explainedbelow. It is assumed here that the vector converter 70 processes MPEG2image codes of frame structure. This is because most of MPEG2 imagecodes are of frame structure.

First, in Step S1, the vector converter 70 determines from, for example,the macro-block type, whether the input motion vector MV pertains to anintra-macro block, an inter-macro block, a skip macro block or no MCmacro block.

If it is determined in Step S1 that the input vector MV pertains to anintra-macro block, the vector converter 70 performs Step S4. In Step S4the vector converter 70 sets the value of 0 to the motion vector mv ofthe 8×8 pixel block that has been subjected to resolution conversion. InStep S4, too, the vector converter 70 sets an intra-mode flag toactivate the motion vector corrector 73. It should be noted that theintra-mode flag is set in an MPEG2 image-encoding system to process anintra-macro block.

If it is determined in Step S1 that the input vector MV pertains to askip macro block, the vector converter 70 performs Step S5. In Step S5the vector converter 70 sets the value of 0 to the motion vector mv ofthe 8×8 pixel block that has been subjected to resolution conversion.

If it is determined in Step S1 that the input vector MV pertains to noMC macro block, the vector converter 70 performs Step S9. In Step S9 thevector converter 70 sets the value of 0 to the motion vector mv of the8×8 pixel block that has been subjected to resolution conversion.

If it is determined in Step S1 that the input vector MV pertains to aninter-macro block, the vector converter 70 performs Step S2.

In Step S2 the vector converter 70 determines whether the 16×16 pixelinter-macro block corresponding the motion vector is one which has framestructure and which should be subjected to frame prediction or one whichhas frame structure and which should be subjected to field prediction.

If it is determined in Step S2 that the inter-macro block has framestructure and should be subjected to frame prediction, the vectorconverter 70 performs Step S6. In Step S6 the vector converter 70converts the motion vector to one that is suitable for frame prediction.

FIG. 9 illustrates the concept of the motion vector conversion that thevector converter 70 carries out when the inter-macro block has the framestructure and should be subjected to frame prediction. Shown in FIG. 9Ais a frame image before the resolution conversion. Shown in FIG. 9B isthe frame image after the resolution conversion. In FIG. 9, pxiindicates an integral pixel before the resolution conversion, hpindicates a half pixel before the resolution conversion, and hpdindicates a half pixel after the resolution conversion. In FIG. 9, thehorizontal and vertical components of a motion vector MV have the samemagnitude of 1 that accords with the half per position of the integralpixel pxi. The magnitudes of the horizontal and vertical components ofthe motion vector MV shall be called, for brevity, “horizontalcomponent” and “vertical component,” respectively.

As described with reference to FIG. 3, the horizontal component of themotion vector subjected to resolution conversion can be obtained fromthe horizontal component of the motion vector MV the image has beforethe resolution conversion, and the rate at which the resolution of theimage has been converted in the horizontal direction. The verticalcomponent of the motion vector subjected to the resolution conversioncan be calculated from the vertical component of the motion vector MVthe image has before the resolution conversion, and the rate at whichthe resolution of the image has been converted in the verticaldirection. Thus, if the resolution in the horizontal direction ischanged to half, the horizontal component of the motion vector will behalf the original value. Similarly, if the resolution in the verticaldirection is decreased to half, the vertical component of the motionvector will be half the original value. In the case shown in FIG. 9, themotion vector, which represents the motion of an image from the positiondm in the preceding frame to the position of a part ob of the image inthe present frame, has a horizontal component of “8” and a verticalcomponent of “12.” On the other hand, the motion vector MV′ (mv)obtained after the resolution conversion has a horizontal component of“4′” and a vertical component of “6′.”

As can be understood from FIG. 9, the components of a motion vector,which are represented by the position of an integral pixel before theresolution conversion, can be represented by the position of an integralpixel or a half pixel even after the resolution conversion. By contrast,the components of a motion vector, which are represented by the positionof a half pixel before the resolution conversion, cannot have anycorresponding pixel after the resolution conversion.

Hence, the components of any motion vector, which are represented by theposition of a half pixel before the resolution conversion, arerepresented after the resolution conversion by the position of a halfpixel of a predicted image. An image signal decoded contains a componentdistorted due to quantization. The image signal may be used as apredicted image. In this case, however, the prediction efficiencydecreases, possibly deteriorating the quality of the image. The imagequality may be prevented from deteriorating if linear interpolation isperformed on the pixels of a reference image at a ratio of 1:1, therebyraising the accuracy of any half pixel. Thus, if the components of anMPEG2 motion vector represent the position of a half pixel, the MPEG2motion vector is converted to an MPEG4 motion vector so that thecomponents of the MPEG4 motion vector may represent the position of thehalf pixel. This enhances the prediction efficiency and preventsdeterioration of the image quality.

Table 1, presented below, shows the components that the motion vectorhas before the motion vector converter 32 converts it and the componentsthat the motion vector has after the motion vector converter 32 convertsit. TABLE 1 Motion vector before conversion, and motion vector afterconversion Remainder of division of vector MV by 4 0 1 2 3 Motion vector[MV/2] [MV/2] + 1 [MV/2] [MV/2] after conversion

In Table 1, [MV/2] is the integral part of the value obtained bydividing the motion vector by 2.

Referring back to FIG. 8, if it is determined in Step S2 that theinter-macro block has frame structure and should be subjected to fieldprediction, the vector converter 70 performs Step S3.

In Step S3, the vector converter 70 determines whether the fieldprediction is top-field prediction or bottom-field prediction.

If the vector converter 70 determines in Step S3 that the fieldprediction is top-field prediction, it performs Step S7. In Step S7, thevector converter 70 operates as will be described below, therebyconverting the motion vector to one that is suitable for top-fieldprediction.

FIG. 10 depicts the concept of the motion vector conversion that thevector converter 70 effects if it is determined in Step S3 that thefield prediction is top-field prediction. Shown in FIG. 10A is a frameimage before the resolution conversion. Shown in FIG. 10B is the frameimage after the resolution conversion. In FIG. 10, pxi indicates anintegral pixel before the resolution conversion, hp indicates a halfpixel before the resolution conversion, and hpd indicates a half pixelafter the resolution conversion. In FIG. 10, the horizontal and verticalcomponents of a motion vector MV have the same magnitude of 1 thataccords with the half per position of the integral pixel pxi.

The horizontal component of the motion vector is converted as is shownin Table 1. Only top fields are extracted, whereby the resolution ischanged to half the original value. Since top-field prediction iscarried out, the vertical component of the motion vector is not changedat all. In other words, the vertical component the motion vector hasbefore the resolution conversion is used as the vertical component themotion vector has after the resolution conversion.

If the vector converter 70 determines in Step S3 that the fieldprediction is bottom-field prediction, it performs Step S8. In Step S8,the vector converter 70 operates as will be described below, therebyconverting the motion vector to one that is suitable for bottom-fieldprediction.

FIG. 11 illustrates the concept of the motion vector conversion that thevector converter 70 effects if it is determined in Step S3 that thefield prediction is bottom-field prediction. Shown in FIG. 11A is aframe image before the resolution conversion. Shown in FIG. 11B is theframe image after the resolution conversion. In FIG. 11, pxi indicatesan integral pixel before the resolution conversion, hp indicates a halfpixel before the resolution conversion, and hpd indicates a half pixelafter the resolution conversion. In FIG. 11, the horizontal and verticalcomponents of a motion vector MV have the same magnitude of 1 thataccords with the half per position of the integral pixel pxi.

As indicated above, only top fields are extracted in the course of theresolution conversion, and the top fields will be used as a referenceimage after the resolution conversion is achieved. It is thereforenecessary to correct the motion vector in time and space, so that thebottom fields used as a predicted image in the MPEG2 image encoding maybe converted to top fields that will be predicted after the resolutionconversion. In the case shown in FIG. 11, the motion vector is correctedin time and space to convert the bottom fields to top fields, thereby toconduct top-field prediction in place of bottom-field prediction. Tostate it more specifically, “1” is added to the vertical component ofthe motion vector. When “1” is added to the vertical component of themotion vector, which has been obtained by means of bottom-fieldprediction, the bottom field is moved upwards by one column (or one row)as is seen from FIG. 11. As a result, the bottom field reaches the samespatial position as a top field. The motion vector becomes similar to amotion vector that may be obtained by means of top-field prediction. Thevertical component of an approximate motion vector MV_(top), which isacquired when the bottom filed that has reached the same spatialposition as the top field is used as a predicted image, is given by thefollowing equation (1):Vertical component: MV_(top)=MV_(bottom)+1   (1)

Note that it is unnecessary to carry out spatial correction on thehorizontal component of the motion vector. The horizontal componentneeds only to be processed in the same way as in the top-fieldprediction.

A time lag occurs between the top field and bottom field of an interlaceimage in the process of MPEG2 image encoding. To eliminate the time lagbetween the bottom field and the top field generated from the bottomfield by means of approximation, the motion vector must be corrected intime.

FIG. 12 shows how the positional relation between fields changes withtime.

As shown in FIG. 12, a time lag of 1 exists between the top field andthe bottom field. In FIG. 12, “a” represents the distance between thebottom field of an I picture and the top field of a P picture. Thedistance “a” is an odd number such as 1, 3, 5, 7, . . . or the like. Ifthe distance “a” is 1, the image will be composed of I, P, P, P, . . .pictures. The vertical component of the motion vector MV′ is given asfollows after the vector MV′ is corrected in terms of time:Vertical component: MV′=(a+1)·approx. MV_(top) /a   (2)

Let us substitute the equation (1) in the equation (2). Then, thevertical component of the motion vector will change to the following,after the motion vector is converted:Vertical component: MV′=(a+1) (MV_(bottom)+1)/a   (3)

The horizontal component of the motion vector after the conversion canbe obtained by multiplying the motion vector before the conversion with(a+1)/a and correcting the motion vector in time, as can be seen fromTable 1.

As mentioned above, the vertical component of the motion vector issubjected to time correction after has been subjected to spatialcorrection. Nonetheless, the vertical component may be first correctedin time and then in space. In this case, the vertical component of themotion vector is given by the following equation (4). The horizontalcomponent of the motion vector will have the same value, no matterwhether it is correct first in space and then in time, or first in timeand then in space.Vertical component: MV′={(a+1)/a}MV_(bottom)+1   (4)

The difference between the values obtained by the equations (3) and (4)is 1/a. Thai is, a difference of 1/a exists between the verticalcomponent found by performing time correction and spatial correction inthe order mentioned and the vertical component found by effectingspatial correction and time correction in the order mentioned.Therefore, the influence the difference imposes depends upon the valueof “a.”

It will be described how the vertical component of the motion vector iscorrected when “a” is 1 and how the vertical component is corrected when“a” is greater than 1, namely, 3, 5, 7, . . . or the like.

How to correct the vertical component of the motion vector when “a” is 1will be explained first.

Substitute 1 for “a” in the equation (3). The vertical component of themotion vector is then given as follows:Vertical component: MV′=2×(MV_(bottom)+1)  (5)

Substituting 1 for “a” in the equation (4), we obtain the followingvertical component of the motion vector:Vertical component: MV′=2×(MV_(bottom)+1)−1   (6)

As a result, the equation (5) provides a value that is an even number,such as 2, 4, 6, . . . or the like if the motion vector MV_(bottom) hasthe value of 0, 1, 2, . . . or the like before it is converted. Thismeans that, once the vertical component has been corrected first inspace and then in time, it is presented at the position of an integralpixel, no matter whether it is presented at the position of an integralpixel or the position of a half pixel before the motion vector isconverted.

If the motion vector MV_(bottom) has the value of 0, 1, 2, . . . or thelike before it is converted, the equation (6) provides a value that isan odd number, such as 1, 3, 5, . . . or the like. In other words, oncethe vertical component has been corrected first in time and then inspace, it is presented at the position of a half pixel, no matterwhether it is presented at the position of an integral pixel or theposition of a half pixel before the motion vector is converted.

Therefore, the vertical component of the motion vector is correctedfirst in space and then in time in order to present the verticalcomponent at the position of an integral pixel, just the same as beforethe motion vector is converted.

On the other hand, the vertical component of the motion vector iscorrected first in time and then in space in order to present thevertical component at the position of a half pixel, just the same asbefore the motion vector is converted.

In summary, in the method of correcting the vertical component, wherein“a” is 1 (a=1), the spatial correction and the time correction arealternately performed on the motion vector yet to be converted.Alternatively, the time correction and the spatial correction areeffected on the motion vector in the order mentioned, or the spatialcorrection and the time correction are effected on the motion vector inthe order mentioned.

It will now be explained how the vertical component of the motion vectoris corrected when “a” is greater than 1 (a≠1), or when “a” is 3, 5, 7, .. . or the like.

If “a” is not 1, that is, if “a” is 3, 5, 7, . . . or the like, thevalue 1/a, i.e., the difference between the vertical component correctedfirst in time and then in space and the vertical component correctedfirst in space and then in time, can be approximated to 0. In this case,it does not matter whether the vertical component is corrected first inspace and then in time or first in time and then in space.

Referring to FIG. 7 again, the vector converter 70 converts the motionvector MV of the input macro block MB that is composed of 16×16 pixelsas described above, thus generating a block composed of 8×8 pixels. Themotion vector mv of the block thus generated is supplied from the vectorconverter 70 to the motion vector adjuster 71.

The motion vector adjuster 71 receives the image-size adjusting flagfrom the external device. In accordance with the flag the motion vectoradjuster 71 adjusts the motion vector mv of the 8×8 pixel block to amotion vector suitable for the size of the image represented by the 8×8pixel block. The motion vector mv thus adjusted is output to the motionvector corrector 73 and the operation section 72.

In other words, the motion vector adjuster 71 adjusts the motion vectormv of the 8×8 pixel block in accordance with the image-size adjustingflag supplied from the external device, so that the pixeladder/extractor 35 may add or extract pixels in the resolution/framerate converter 31. The motion vector adjuster 71 outputs the motionvector mv thus adjusted.

How the motion vector adjuster 71 performs its function will bedescribed in detail, with reference to FIG. 13.

The motion vector adjuster 71 receives the data representing the size ofthe image input. From this data the motion vector adjuster 71 determinesin Step S11 whether the image output from the MPEG2 decoder 30 has aresolution of m×n pixels and whether the m/2 pixels arranged in eachcolumn and n/2 pixels arranged in each row are both multiples of 16. Ifit is determined in Step S11 that m/2 pixels and the n/2 pixels are bothmultiples of 16, the switching section SW3 is turned on, outputting theMPEG4 motion vector mv (not corrected yet) that has been supplied fromthe motion vector converter 32. If it is determined in Step S11 thatboth m/2 pixels and the n/2 pixels are not multiples of 16 (that is, ifthere is a remainder of 8 pixels), the motion vector adjuster 71performs Step S12.

In Step S12, the motion vector adjuster 71 determines, from theimage-size adjusting flag, whether eight pixels (i.e., the remainder ofthe division by 16, which the resolution/frame rate converter 31 hasperformed) have been extracted or not. If YES, the switching section SW4is turned on. In this case, the motion vector adjuster 71 outputs themotion MPEG4 vector mv of each block b, which does not corresponds tothe eight pixels that have been extracted. If NO in Step S12, the motionvector adjuster 71 carries out Step S13.

In Step S13, the motion vector adjuster 71 controls the switchingsection SW5 in accordance with the image-size adjusting flag. The motionvector mv of each block b that corresponds to the eight pixels added inthe resolution/frame rate converter 31 is thereby set at “0.” (That is,the movable contact of the switching section SW5 is connected to the “0”stationary contact.) A motion vector of “0” is thereby output for theeight pixels added, from the motion vector adjuster 71. The motionvector adjuster 71 outputs the motion vectors mv of the other blocks b,without processing them at all.

Referring back to FIG. 7, the motion vector mv thus adjusted by themotion vector adjuster 71 is supplied to the operation section 72 andthe motion vector corrector 73.

The operation section 72 performs the operation of the followingequation (7) by using the motion vector mv, thereby obtaining an MPEG4motion vector MVT for a 16×16 pixel macro block MBT that consists offour 8×8 pixel blocks b having motion vectors mv1 to mv4, respectively.${MV}_{T} = \left\{ \begin{matrix}0 & {{ifthefourblocksareallintra}\text{-}{blocks}} \\\frac{\sum\limits_{i = 0}^{3}{mvi}}{{{No}.{ofmv}} \neq 0} & {{ifaleastoneofthefourblocksisaninter}\text{-}{block}}\end{matrix} \right.$

That is, the operation section 72 finds the sum of the motion vectorsmv1 to mv4 of the four blocks b1 to b4 converted from a non-intra macroblock MB that is one of the four blocks converted from an MPEG2 macroblock MB. The sum is divided by the number of blocks that have beenconverted from a non-intra macro block. The operation section 72 outputsthe quotient of the division as the motion vector MV_(T) of the 16×16pixel macro block MB_(T) that accords with the MPEG4 image encoding. TheMPEG4 motion vector MV_(T) of the 16×16 pixel macro block MB_(T) may bedetermined from the mean weight applied to the DCT coefficient of themotion vector mv of the 8×8 pixel block b.

The MPEG4 motion vector MV_(T) output from the operation section 72 issupplied to the motion vector corrector 73.

The motion vector corrector 73 receives any one of the motion vectors mvof the 8×8 pixel blocks b supplied from the motion vector adjuster 7,that corresponds to a MPEG2 intra-macro block. The motion vectorcorrector 73 replaces the motion vector mv with the motion vector MV_(T)the operation section 72 has found for the 16×16 pixel macro blockMB_(T).

FIG. 14 illustrates the structure of the motion vector corrector 73 indetail.

As shown in FIG. 14, the motion vector corrector 73 comprises a switchSW2 and a vector-replacing section 80. The switch SW2 has a movablecontact and two stationary contacts a and b. The switch SW2 receives themotion vector mv of an 8×8 pixel block b which has been output from themotion vector adjuster 71 and which accords with the image size. Aninfra-mode flag is supplied to the switch SW2. In accordance with theinfra-mode flag the movable contact of the switch SW2 is connected toeither the first stationary contact a or the second stationary contactb.

If the infra-mode flag is set, indicating that the input motion vectormv corresponds to an MPEG2 intra-macro block, the movable contact willbe connected to the first stationary contact a. The motion vector mv isthereby supplied to the vector-replacing section 80.

The vector replacing section 80 receives the motion vectors mv1 to mv4of the four 8×8 pixel blocks b1 to b4 converted from the MPEG2intra-macro block. The section 80 also receive the motion vector MV_(T)of the 16×16 pixel macro block MB_(T) which has been generated by theoperation section 72 and which accords with the MPEG4 image encoding.The section 80 replaces the motion vectors mv1 to mv4 with the motionvector MV_(T). The MPEG4 motion vector is output from the vectorreplacing section 80. Alternatively, the motion vectors mv1 to mv4 maybe replaced by a motion vector converted from the motion vector of aninter-macro block located near the intra-macro block or by a motionvector converted from the inter-macro block located nearest theintra-macro block. Further, the motion vector may be “0.” If the motionvectors of the four 8×8 pixel intra-macro blocks b1 to b4 have beenconverted from intra-macro blocks, they will be “0.” In this case, themotion vector MV_(T) of the 16×16 pixel macro block MB_(T) is “0,” too.The motion vector for use in the MPEG4 image encoding is therefore “0,”and the macro block changes to an intra block.

If the infra-mode flag is not set, indicating that the input motionvector mv does not correspond to an MPEG2 intra-macro block, the movablecontact is connected to the second stationary contact b. In this case,the input motion vector mv of the 8×8 pixel block b is output from themotion vector corrector 73.

Referring back to FIG. 7, the motion vectors mv1 to mv4 of the 8×8 pixelblocks b1 to b4, which have been output from the motion vector corrector73, are output, together with the motion vector MB_(T) that consists of8×8 pixel blocks b1 to b4, which have been generated by the operationsection 72. The motion vectors mv1 to mv4 and the motion vector MB_(T)are supplied to the MPEG4 image encoder 33 shown in FIG. 2.

The MPEG4 image encoder 33 performs MPEG4 image encoding on the imagedata output from the resolution/frame rate converter 31, by using theMPEG4 motion vector output from the motion vector converter 32. As aresult, the MPEG4 image encoder 33 generates a bit stream of MPEG4 imagecodes.

As has been described above, the first embodiment of the inventionconverts a bit stream of MPEG2 image codes to a bit stream of MPEG4image codes. Nonetheless, the present invention is not limited to thefirst embodiment. Rather, the invention can be applied to an apparatusthat converts an input image signal to an image signal of the sameformat as the input image signal. Moreover, the invention can be appliedto an apparatus that converts an input image signal to an image signalhaving a resolution and frame rate different from those of the inputimage signal.

FIG. 15 shows an image-coding format converting apparatus that is thesecond embodiment of this invention. The components identical or similarto those of the first embodiment (FIG. 2) are designated at the samereference numerals in FIG. 15 and will not be described in detail.

The image-coding format converting apparatus shown in FIG. 15, i.e., thesecond embodiment, is identical in basic structure to the firstembodiment that is shown in FIG. 2. The MPEG2 image decoder 30 decodesthe variable-length MPEG2 image codes of the input bit stream andextracts the motion vector of the P picture only and the parameters suchas the macro-block type and the like. The motion vector and theparameters are output from the MPEG2 image decoder 30 to the motionvector converter 32. The MPEG2 image codes are decoded and the motionvector and parameters are extracted in the same way as in the firstembodiment shown in FIG. 2. The second embodiment comprises a switchSW6, a frame memory 154 and a motion compensation predictor 155. Theswitch SW6 is connected to the output of the motion vector converter 32and to the input of the frame memory 154. The motion compensationpredictor 155 is connected to the output of the frame memory 154.

The switch SW6 comprises a movable contact and two stationary contacts aand b. The movable contact is connected to the output of the motionvector converter 32. The first stationary contact a is connected to theframe memory 154. The second stationary contact b is connected to theMPEG4 image encoder 33. The movable contact of the switch SW6 isconnected to the first stationary contact a only when the motion vectorsmv1 to mv4 of four 8×8 pixel MPEG4 blocks b1 to b4, output from themotion vector converter 32, correspond to an MPEG2 infra-macro block.When the movable contact is connected to the second stationary contactb, the apparatus operates in the same manner as the apparatusillustrated in FIG. 2.

When the movable contact of the switch SW6 is connected to the firststationary contact a, the MPEG4 motion vector mv output from the motionvector converter 32, the image signal that has been output from theresolution/frame rate converter 31, and the image signal correspondingto the bit stream output from the MPEG4 image encoder 33 are stored intothe frame memory 154.

In the second embodiment, the motion vector of the 16×16 pixel macroblock and the motion vector of the 8×8 pixel block are detected beforethey are supplied to the MPEG4 image encoder 33. Therefore, of the imagesignal stored in the frame memory 154, the image signal supplied fromthe resolution/frame rate converter 31 represents the present frameimage, and the image signal supplied from the MPEG4 image encoder 33represents a prediction-reference frame image. The motion compensationpredictor 155 uses the present frame image and the prediction-referenceframe image, thereby detecting an MPEG4 motion vector that pertains tothese frame images. The prediction-reference frame image may berepresented by an image signal output from the resolution/frame rateconverter 31. Further, the image signals output from theresolution/frame rate converter 31 may alternately represent the presentframe image and the prediction-reference frame image. The MPEG4 motionvector output from the motion compensation predictor 155 is supplied tothe MPEG4 image encoder 33.

FIG. 16 shows an image-coding format converting apparatus that is thethird embodiment of the present invention.

As illustrated in FIG. 16, this image-coding format converting apparatuscomprises an image pre-processing filter 161, a motion compensator 162,an MPEG2 image encoder 163, a motion vector converter 164, an MPEG4encoder 165, and a resolution/frame rate converter 166. The imagepre-processing filter 161 receives an image signal from an imagingdevice 160 or an image-receiving device 167. The imaging device 160 has,for example, a CCD (solid-state imaging element). The image-receivingdevice 167 is, for example, a tuner or the like. The imagepre-processing filter 161 performs, if necessary, pre-processing on theinput image signal, thereby removing noise from the image signal. Theimage signal is supplied from the image pre-processing filter 161 to themotion compensator 162, MPEG2 image encoder 163, motion vector converter164 and resolution/frame rate converter 166. If the image signal neednot be pre-processed at all, the image pre-processing filter 161 can bedisposed of.

The motion compensator 162 receives frame-rate/resolution data A from anexternal device. The motion compensator 162 calculates, from theframe-rate/resolution data A, a motion vector of each frame image signalsupplied from the image pre-processing filter 161. The motion vectorcalculated is supplied to the MPEG2 image encoder 163 and motion vectorconverter 164.

The MPEG2 image encoder 163 encodes the image signal supplied from theimage pre-processing filter 161, by using the motion vector output fromthe motion compensator 162. Thus, the MPEG2 image encoder 163 generatesa bit stream of MPEG4 image codes.

The motion vector converter 164 operates in the same way as the motionvector converter 32 incorporated in the first embodiment. The converter164 converts a motion vector in accordance with theframe-rate/resolution data A supplied from the device, therebygenerating a new motion vector. The new motion vector is supplied to theMPEG4 encoder 165.

The resolution/frame rate converter 166 operates in the same way as theresolution/frame rate converter 31 provided in the first embodiment.That is, the resolution/frame rate converter 166 effectsresolution/frame rate conversion on the image signal output from theimage pre-processing filter 161, in accordance withframe-rate/resolution data A and frame-rate/resolution data B, bothsupplied from the external device. Thus, the resolution/frame rateconverter 166 generates an image signal, which is supplied to the MPEG4encoder 165.

The MPEG4 encoder 165 receives the image signal generated by theresolution/frame rate converter 166. The MPEG4 encoder 165 encodes theimage signal in accordance with the motion vector output from the motionvector converter 164. The encoder 165 generates a bit stream of MPEG4image codes.

In the third embodiment, the motion vector converter 164 is used for onemotion compensator 162. The motion vector the motion vector converter164 generates can be therefore used in both the MPEG2 encoder 163 andthe MPEG4 encoder 165. The amount of data processed in the thirdembodiment is smaller than the amount of data that is processed in theconventional apparatus that needs two motion compensators.

1. An apparatus for converting a first encoded image signal to a secondencoded image signal, said apparatus comprising: signal-decoding meansfor decoding the first encoded image signal, thereby outputting an imagesignal and a first motion vector used in decoding the first encodedimage signal; signal-converting means for generating a second imagesignal from the image signal by resolution converting and frame rateconversion; motion-vector converting means for generating the secondmotion vector from the second image signal based on the first motionvector; and signal-encoding means for encoding the second image signalinto the second encoded image signal, by using the second motion vector.2-3. (canceled)
 4. The apparatus according to claim 1, wherein thesignal-converting means adds pixels to, or extracts pixels from, theimage signal. 5-6. (canceled)
 7. A method of converting a first encodedimage signal to a second encoded image signal, said method comprisingthe steps of: decoding the first encoded image signal, therebyoutputting an image signal and a first motion vector used in decodingthe first encoded image signal; generating a second image signal fromthe image signal by resolution converting and frame rate conversion;generating the second motion vector from the second image signal basedon the first motion vector; and encoding the second image signal intothe second encoded image signal, by using the second motion vector. 8-9.(canceled)
 10. The method according to claim 7, wherein pixels are addedto or extracted from the image signal in the step of converting theimage signal to a second image signal. 11-12. (canceled)